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ABSTRACT 

Traditional professional development is often characterized as being 
expensive, time consuming, and lacking impact. In contrast, online profes- 
sional development provides greater flexibility and is becoming increas- KEYWORDS 

ingly popular for school personnel. In this article, we report the process Online learning; qualitative 
and outcomes of gathering feedback to adapt traditional in-person to methods; teacher education 
online training differentiated for the participants to maximize utility, effi- 

ciency, and effectiveness in meeting the needs of key stakeholders. Focus 

groups were conducted with teachers, administrators, and other specialists 

to gather feedback on content as well as how online learning modules 

(OLMs) could be tailored to meet specific school site and team needs. We 

discuss our findings in the context of continuing efforts to improve general 

and specific professional development opportunities. 


Introduction to the problem 


Professional development (PD) occurs at all levels of education and refers to a variety of special- 
ized learning experiences intended to help administrators, teachers, and other educators improve 
their practices and support student learning (Darling-Hammond, Hyler, & Gardner, 2017a, 
2017b, 2017c; Desimone & Garet, 2015; Simonsen et al., 2019). PD can be delivered in person 
and/or online with participation in virtual experiences (Bahr et al., 2019; Brennan, Sellmaier, 
Jivanjee, & Grover, 2019). The structural features and characteristics of PD vary by the focus of 
the activities (i.e., type of content, type of learning, and coherence or contextual fit with partici- 
pants’ professional goals and needs) as well as by organization (e.g., workshop, conference), dur- 
ation (i.e., total hours as well as span of time), and participation (e.g., groups, teams, individuals; 
Birman, Desimone, Porter, & Garet, 2000) . In many local education agencies, PD is used to sup- 
port skill acquisition as well as innovative and ongoing practices; and, models of PD vary across 
districts, with some schools using periodic workshop-style trainings and others using ongoing 
embedded activities (Brennan et al. 2019; Darling-Hammond et al. 2017c; Kraft, Blazar, & 
Hogan, 2018; Yoon, Duncan, Lee, Scarloss, & Shapley, 2007). Although research is mixed around 
the overall effectiveness of PD (Darling-Hammond et al., 2017a; Sumsion et al., 2015), it is gener- 
ally agreed that if designed and implemented well, it can improve teaching and strengthen practi- 
ces school-wide (Birman et al., 2000; Darling-Hammond et al., 2017a, 2017b, 2017c; Garet, 
Porter, Desimone, Birman, & Yoon, 2001; Gast, Schildkamp, & van der Veen, 2017). 
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After a review and comprehensive analysis of 35 studies that reported positive links among 
teacher professional development, teaching practices, and student outcomes, Darling-Hammond 
et al., 2017a, 2017b, 2017c) reported that effective PD is focused on specific content that is deliv- 
ered in active learning sessions that support collaboration, use models of effective practice, pro- 
vide coaching and other support, offer feedback, and include adequate time to learn, practice, 
and evaluate strategies expected to bring about changes in practice. We designed our professional 
development to include these features. Specifically, we focused on a singular content-based prac- 
tice (ie., problem solving) and used interactive, collaborative activities and other strategies to pro- 
vide contextualized professional learning. We also included coaching, sharing, and other feedback 
opportunities to extend the time to learn beyond the online learning sessions. 


Team-initiated problem solving (TIPS) 


Problem solving and data-based decision making and problem solving is a core feature of frame- 
works that state education agencies, local education agencies, and schools use to support imple- 
mentation of practices such as Multi-Tiered System of Support (MTSS), Response to Intervention 
(RTI), and Positive Behavior Interventions and Supports (PBIS) (cf. Horner, Sugai, & Anderson, 
2010; Horner, Sugai, & Lewis, 2015; McCart, Sailor, Bezdek, & Satter, 2014; National Center on 
Intensive Intervention, 2019; National Center on Response to Intervention, 2010; OSEP Technical 
Assistance Center on Positive Behavioral Interventions & Supports, 2015); and, more than 25,000 
schools across the United States are using PBIS (OSEP Technical Assistance Center on Positive 
Behavioral Interventions & Supports, 2019). A variety of models for school-based problem solving 
have been proposed (cf. Bransford & Stein, 1984, 1993; Deno, 2005; Fuchs & Deshler, 2007: 
Leithwood, 2010; Tilly, 2008); however, research and evaluation findings provide an equivocal 
picture of the extent to which these approaches are effective or used in schools ( Balu et al., 2015; 
Burns, Peters, & Noell, 2008; Burns & Symington, 2002; Burns, Vanderwood, & Ruby, 2005; 
Crone et al., 2016; Fixsen, Blase, Metz, & Van Dyke, 2013; Fuchs & Fuchs, 2017). One example 
of an effective data-based decision making model used in schools implementing PBIS is the 
Team-Initiated Problem Solving (TIPS) framework (Newton, Algozzine, Algozzine, Horner, & 
Todd, 2011; Newton, Todd, Algozzine, Horner, & Algozzine, 2009). 

TIPS is a six-stage framework of data-based decision making that has been used across tiers 
within implementations of Positive Behavior Interventions and Supports (PBIS) and validated in 
two randomized waitlist controlled trials (Horner et al., 2018; Newton, Horner, Algozzine, Todd, 
& Algozzine, 2012). The success of teams implementing TIPS in these studies, under conditions 
of rigorous experimental control, establishes TIPS as an evidence-based practice for data-based 
decision making within education. TIPS professional development typically occurs in a single 
full-day workshop format, with all members of a given team present and participating (Newton 
et al., 2009; 2011; Todd et al., 2011). In typical training, teams apply what they have learned in a 
mock team meeting near the end of the day; and, follow-up coaching for the next two team meet- 
ings is an integrated component of TIPS PD (Newton et al., 2009). 


Features and characteristics of TIPS PD 

Within the TIPS framework, teams first establish foundations for effective meetings, including 
establishing roles, setting and adhering to an agenda for each meeting, and setting and adhering 
to a schedule to when team meetings will occur. Roles within TIPS include Facilitator, Minute 
Keeper, and Data Analyst. Within a TIPS data meeting, the Facilitator guides the team through 
the agenda (established and disseminated at least one day before the meeting). The Minute 
Keeper records critical information from the team’s problem solving and data analysis discussions 
onto the TIPS Meeting Minutes form, or similar, for future tracking and follow-up. The Data 
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Analyst summarizes and interprets data for the team during the meeting and ensures all relevant 
data will be available for the meeting, at least two days prior. TIPS meetings are efficient, begin- 
ning and ending within 10 minutes of the scheduled times, and prompting the team to review the 
most critical information for rendering and evaluating decisions in a time-sensitive manner. The 
meeting foundations serve as context within which effective decision making occurs when apply- 
ing the TIPS problem solving framework. 

The TIPS problem solving framework includes six stages: (a) identifying a problem with preci- 
sion, (b) identifying a goal for change, (c) identifying a solution with a plan for contextual fit, (d) 
implementation of solution, (e) monitoring impact of the solution, and (f) making a summative 
evaluation decision. Within the TIPS framework, a goal must specify what change will occur on a 
given indicator of performance, and by what date the change will occur. Once an appropriate 
goal has been identified, the team moves onto selecting a solution that it anticipates will meet the 
specified goal for those involved. Solutions emphasize environmental changes and malleable fac- 
tors associated with instruction. Solutions are discussed and a solution that the team deems feas- 
ible in the local context is selected implementation. Once identified, a plan is then articulated for 
how the solution will be implemented in practice for those affected. Specifically, who will be 
responsible for implementing the proposed changes/intervention, when this will occur, with what 
frequency it will be implemented, and when it will be delivered during the school day (matched 
to the when of the problem statement for problems involving social behavior). Following specifi- 
cation of the solution implementation plan, the team decides how implementation will be moni- 
tored for fidelity and by whom. 

For problems which have been identified, a goal set, a solution selected and implemented, and 
fidelity monitored, the team will review student progress and fidelity of implementation data at sub- 
sequent meetings. This portion of the meeting coincides with the remaining stages of the TIPS 
problem-solving process. The implementation of the solution will be monitored using fidelity data 
collected according to the previous problem-solving session plan, and impact of this implementa- 
tion on relevant student performance will be evaluated. Once this has occurred, a summative evalu- 
ation decision will be made regarding the effectiveness of a particular intervention for a particular 
student or group of students. During the evaluation of intervention impact on student performance 
(after confirming implementation was carried out with fidelity), the team may determine that the 
intervention had no effect, it had a detrimental effect, performance improved but not to criterion, 
performance improved to criterion (goal is met), or that the data are unclear. Based upon this 
evaluation the team will then make a summative evaluation decision. If improvement occurred but 
not to criterion, the team will either identify a malleable factor to adjust or decide to defer further 
action and see if the impact increases during the next data collection cycle. If the goal has been 
met, the intervention will likely be reduced in intensity, or potentially discontinued altogether. If 
performance has not improved, the intensity will be increased or another malleable factor will be 
altered, potentially including trial of a completely different intervention. 

Typical training moves systematically across each component and stage (e.g., meeting founda- 
tions, identifying a problem with precision, identifying a goal) of the process and the design and 
structure of the TIPS model create opportunities for active, collaborative, and effective profes- 
sional development that can be delivered in content-specific modules of effective practices. The 
organization and flow also create opportunities for frequent feedback, reflection, and support 
available in online learning. 


Online learning 


Online learning is widely used technology (Martin, Ahlgrim-Delzell, & Budhrani, 2017; Opfer & 
Pedder, 2011; Shaha & Ellsworth, 2013; Shirrell, Hopkins, & Spillane, 2019; Waitoller & Artiles, 
2013; Yoon et al., 2007). While many schools rely on in-person experiences to support professional 
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development of administrators, teachers, and other staff, many also use online learning to build and 
maintain skills (Blitz, 2013; Brennan et al., 2019; Cook & Steinert, 2013; Dash et al., 2012; Glover, 
Reddy, Kurz, & Elliott, 2018; Kleiman et al., 2000; Mohr & Shelton, 2017; Rice, 2017). Online PD 
can provide a broader level of content and increase accessibility, especially for schools in rural areas 
and for specific program content (Smith & Sivo, 2012). In the traditional model of in-person pro- 
fessional development, many issues can arise with providing ongoing training for teachers. Schools 
often lack adequate time and resources to provide training as most PDs are delivered after school 
or in the summer before, both of which are also needed for planning and addressing issues during 
the school year (Wei, Darling-Hammond, & Adamson, 2010; Yoon et al., 2007). Many teachers 
express frustration because of time conflicts, a lack of relevancy, and little choice in what profes- 
sional development they receive (Dede, Breit, Ketelhut, McCloskey, & Whitehouse, 2005; Dede, Jass 
Ketelhut, Whitehouse, Breit, & McCloskey, 2009). Online learning provides one avenue to mitigate 
some of these challenges by having more flexibility around where and when teachers can engage as 
well as by increasing the accessibility of content to more schools. 

In addition to these general challenges, schools often face difficulty with tailoring profes- 
sional development to meet the diverse and changing needs of their staff (Brennan et al., 2019; 
Desimone, Porter, Garet, Yoon, & Birman, 2002; Garet et al., 2001; Glover et al., 2018; 
Koellner & Jacobs, 2015; Peterson-Ahmad, Hovey, & Peak, 2018; Porter, Garet, Desimone, 
Yoon, & Birman, 2000). Schools likely have staff teams with a variety of experiences and 
responsibilities that could necessitate different professional development content according to 
teachers’ experiences or roles in the school. For example, a seasoned reading intervention 
teacher will have different professional development needs from a first-year educational assist- 
ant working in a behavior focused classroom. In addition, school teams, such as the positive 
behavior intervention and support (PBIS) team or any of the data-based decision making 
teams in a multi-tiered support system (MTSS), will have members from both general and spe- 
cial education and will have team members who change year to year. This fluctuation necessi- 
tates both differentiated and targeted training for the various team members according to their 
role, experience, and familiarity with the content (Darling-Hammond et al., 2017; Simonsen 
et al., 2019; Sumsion et al., 2015). Online learning provides schools with the option to tailor 
professional development to fit the diverse needs of staff members and to respond to adjust- 
ments in team make-ups, which cause teachers to need to quickly acquire specific content 
knowledge to fully participate in their teams (Cook & Steinert, 2013; Mohr & Shelton, 2017; 
Rice, 2017; Shaha & Ellsworth, 2013). 

Online learning has shown to be an effective tool for delivering teacher professional develop- 
ment and increasing teacher knowledge (Dash et al., 2012; Glover et al., 2018). Specific features 
have been identified as influencing the effectiveness of online learning for teachers. In general, 
ensuring active learning by using various modes of engagement as well incorporating a social 
component have both been found to enhance PD effectiveness for teachers (Bayar, 2014; Shaha & 
Ellsworth, 2013; Williford et al., 2017). Additionally, teacher acceptability has also been shown to 
mediate the overall effectiveness of online learning and should be considered when developing 
and testing online learning tools (Dede et al., 2005, 2009). Teacher acceptability can be increased 
through incorporating a social element, such as supporting conceptual change, having online 
learning communities, or interacting around professional development material with colleagues 
(Bayar, 2014; Dash et al., 2012; Dede et al. 2005, 2009; Englund, Olofsson, & Price, 2017; 
Kleiman et al., 2000; Mohr & Shelton, 2017; Rice, 2017; Smith & Sivo, 2012). While online learn- 
ing has shown to be an effective tool for increasing teacher knowledge, questions remain about 
how to best use the technology in building and supporting effective teaching practices and the 
extent to which stakeholder perceive key features of OLMs as more or less supportive of its effi- 
cient and effective use. 
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Purpose of the study 

TIPS has been shown to be effective in improving school-based decision making. We were inter- 
ested in developing a TIPS online learning module (OLM) to support the increasing need for and 
interest in TIPS PD. In this article, we describe the use stakeholder feedback during the initial 
stages of an iterative TIPS OLM development process. Specifically, we report on the process and 
outcomes of gathering feedback to improve the user experience and ensure the online tool is use- 
ful, efficient, and effective in meeting the needs of key stakeholders. 


Method 


Participants and setting 

Nineteen school staff members from five schools participated in two focus groups during the 
spring semester of the school year. Each of these five schools were from a single medium-sized 
school district in the Pacific Northwest and had been implementing PBIS for the past four years. 
Students in this district are predominantly White (67%), and most are economically disadvan- 
taged (~75%). Approximately 23% are Hispanic, 7% Multi-Racial, and 1-2% each of American 
Indian/Alaska Native, Black/African American, Hawaiian/Pacific Islander, and Asian. 
Approximately 11% of students in the district are English Learners, and approximately 17% have 
an individualized educational plan. On most current assessments, roughly 55% of elementary and 
middle school students performed below state grade level expectations in reading and 60-70% 
performed below state grade level expectations in mathematics. 

Participants for the present study were recruited using a purposive sampling procedure (Berg 
& Lune, 2012) with the goal of collecting the perspectives of school staff members who serve on 
their respective schools’ Tier 2 PBIS. To establish a range of perspectives and reflect typical (cf. 
Taie & Goldring, 2017) and local distributions of settings, professionals from four elementary and 
one middle school were included. Further, participants’ roles within their schools were general or 
special education teacher (n= 6, 32%), school psychologist (n= 6, 32%), support personnel (e.g., 
Behavior Specialist, Educational Assistant, n=5, 26%), or administrator (n=2, 10%), and were 
representative of typical distributions of professionals on school-based teams. All participants 
were members of their respective schools’ Tier 2 PBIS teams and had some experience using the 
TIPS model. 

This phase of project took place in a conference room, approximately 20’ x 30’, on a university 
campus. A projector was used to display information to orient all participants. For individual 
activities laptops were available for those without a computer or for whom using their personal 
computer was not feasible. Participant seats were arranged in a large ‘U’ configuration in the 
middle of the room. 


Procedure 


The online training module on TIPS used in the present study focused on one step of the TIPS 
model: precision problem statements. The training module is anticipated to take approximately 
60 minutes to complete, however due to time constraints participants were given 45 minutes to 
complete the module therefore several of the participants did not finish. In light of the explora- 
tory nature of the study, this was not considered a significant limitation of the study. Precision 
problem statements are one of six steps in the TIPS problem-solving process and form the basis 
for the remaining process-oriented components. Precision problem statements require specifica- 
tion of six elements: (a) who the problem affects, (b) what the problem is (and to what degree), 
(c) when the problem is occurring, (d) where the problem is occurring, (e) how often the prob- 
lem is occurring, and (f) why the problem is thought to occur. The module presented during the 
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present study contained these six elements as well as an additional element focused on goal set- 
ting related to this type of problem statement. 

The TIPS OLM was created through the online learning management system Obaverse which 
was developed at the University of Oregon. Oba seamlessly combines and extends the leading 
open source technologies to provide a powerful, flexible, learner-centric hub for both commun- 
ities of practice and online learning. To use the technology, users login on the homepage and are 
presented with a list of educational modules. They can then navigate independently through the 
different modules. For TIPS, each module covers a different component of the TIPS models. User 
can select relevant content and navigate between modules as desired. 

Each segment of the module began with a video presentation of core content related to the 
component being presented. Embedded in each video were checks for understanding immedi- 
ately after critical portions of the component had been discussed. Members of the research 
team circulated throughout the duration of the exploration phase of the focus group (when 
participants were exploring the OLMs) to answer questions and assist participants as necessary. 
Following these videos, interactive practice activities were presented to give participants an 
opportunity to apply the content just taught in a scaffolded context with performance feed- 
back. Scaffolding was withdrawn as participants demonstrated increasing proficiency during 
these activities and were able to apply the skills associated with presented content 
independently. 


Data collection 

We conducted focus groups (Buchanan, Nese, & Clark, 2016; Cyr, 2016; Patton, 1999) where the 
principal investigator and co-principal investigator served as the facilitators and two trained graduate 
students served as data collectors. A semi-structured format was used to document opinions of 
many individuals across two different days, to assess levels of consensus across participants, and to 
provide an opportunity for new questions to be revealed. This approach provided participants with 
the opportunity to interact with one-another and engage in meaningful discussion of their experi- 
ence with the content and platform. It also produced more detailed and authentic information than 
would have been likely through a strictly researcher-driven process (Morgan, 1996). 

The research team developed information and open-ended guiding questions (see Appendix 
A) which were used by both the facilitators and the data collectors to organize group discussion 
and elicit feedback pertaining to the clarity of content delivery and ease of navigation through 
the online TIPS training modules. The following content was included: (a) a description of the 
study, (b) introductions and discussions of prior experiences with online learning and the TIPS 
model (e.g., “What have you liked/disliked about online learning in the past?”), (c) questions 
about the types of training and support activities that have been most useful in applying the 
TIPS model (“What lessons have you learned over the years?”), (d) questions about the types of 
training or supports that would be most useful for new team members learning the TIPS model 
(e.g., “What are some things you wish you would have been taught when first using data in 
data team meetings?”), (e) details about a proposed online training course on TIPS, and (f) 
questions about the fit of the proposed training course. The focus groups were structured such 
that participants were introduced to the general purpose of the meeting and received an over- 
view of the content of the course prior to working through some of the modules, and then, 
reconvened as a group for course-specific discussion and feedback. Data collectors independ- 
ently took notes on both positive and constructive feedback provided by participants and this 
information served as the unit of analysis for decision making related to how to improve the 
user experience and ensure that the planned tool is useful, efficient, and effective in meeting 
the needs of key stakeholders. 
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Design and data analysis 


In the context of an exploratory case study (Yin, 2017), a qualitative approach involving sequen- 
tial focus groups was used to gather feedback from key stakeholders for use in improving the 
experience and content of the OLM. This design is often used as a prelude to provide formative 
information for subsequent research and/or product development (Hancock & Algozzine, 2017). 
As previously noted, two trained data collectors independently recorded participant feedback dur- 
ing the focus groups, then met to promote reliability in their collective notes across two specific 
aspects of the OLM. First, feedback was summarized with respect to the content presented in the 
OLM, how easy it was to understand, and any prerequisite knowledge needed in order to improve 
understandability of the content. Second, feedback was summarized with respect to the online 
platform including specifics about the look, feel, and functionality of the module. The data collec- 
tors then coded feedback as either positive or constructive, and the full research team met to 
review all feedback and make decisions about edits and adaptations to make based on the sugges- 
tions from the focus group participants. Prior to the second focus group session, feedback was 
used to adjust content presentation, formatting, and structure of the modules as feasible. This 
process was then repeated during and following the second focus group session, with changes 
implemented before the next phase of development work and pilot testing. 


Results 


There were numerous points of consensus evident in the individual responses and field notes col- 
lected and analyzed across individuals and focus groups. Overall, feedback from the first group 
emphasized the online delivery platform and structural aspects of the presentation training con- 
tent. The second group also generated feedback related to content and sequencing and confirmed 
that the issues raised during the first group had been satisfactorily addressed. Additional findings 
are detailed in the following sections. 


Feedback on OLM content 


Strengths 

Members of both focus groups provided feedback related to the content of the online training 
(see Table 1). All participants worked through the modules and offered feedback. More than 90% 
of the participants provided unique and substantive feedback, with several perspectives repeated 
or endorsed by multiple participants. Feedback on the content of the online TIPS training was 
varied and included observations pertaining to specific aspects of individual items as well more 
general observations about the approach to content delivery and the broader scope of the 


Table 1: Examples of Participants Feedback related to Content of Online Training. 
Focus Feedback 


Strengths “The sequence of instruction was laid out well. Instruction then practice then feedback, that’s good 
for new learners.” 
“| liked the six question check on precision problem statements ... and the feedback on the question 
| missed with an opportunity to retry was helpful and motivating.” 
“It's great that there are links to research articles because teachers can’t always access those.” 
Improvements “Having a pre-test would be very useful for helping identify needed supports and which modules to 
send staff... feedback on completion would be helpful too.” 
“Interaction with peers has been critical for professional learning... online delivery currently lacks 
this ... more exercises and practice might help alleviate this.” 
“Short clips for staff meetings would be great-examples: what is function of behavior? Why kids go 
to CICO? Little vignettes for all staff to get important content but not have to do the 
whole module.” 
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training. Focus group participants particularly appreciated the scope of content and the efficiency 
of its delivery and identified multiple strengths including usage of a clear and consistent sequence 
of instruction with both opportunities to practice and provision of meaningful feedback. They 
also commented that instructional sequences were appropriately focused and very engaging, with 
feedback that drove deeper engagement than might otherwise be expected. Beyond simply the 
variety and depth of interactive methods in the TIPS OLM, users specifically appreciated the 
match between specific content and relevant interactive approaches to content delivery and prac- 
tice. Comments and responses also suggested that the overall scope and pacing were appropriate 
and participants believed that this would facilitate faster learning by covering content in appropri- 
ate depth without spending too much time on any one topic. 


Areas of improvement 

Focus group participants also provided feedback on challenges and potential areas for improvement 
in the content delivered. Feedback pertained to a full range of content levels from specific items 
through modules and including the scope of the training overall. The feedback for improving the 
overall training in terms of content pertained to participants’ desire to have the ability to target the 
training for specific audiences, either through selecting which materials were delivered to whom, or 
through the inclusion of short-form videos that would be of interest to a broader school staff audi- 
ence. Users believed that differentiated content options would maximize training utility within 
schools. The remaining feedback pertaining to content of the training was related to specific items 
or shorter segments of content. Participants identified several instances where content was delivered 
with less clarity, either due to phrasing during presentation or the mode of feedback following inter- 
active practice. Specifically, users identified a need for more clarity in the presentation of content 
related to the TIPS process across tiers. They also expressed some confusion around how content 
described at a given tier applied to the TIPS process at other tiers and expressed a desire for clearer 
transitions between content related to different tiers. The section on precision problem statements 
was seen as an area where content was difficult to generalize across tiers. Participants also reported 
some areas where the content was overly dense, and thus difficult to understand, especially for new 
team members. It is worth noting again that participants were representatives from school teams 
with different histories of implementation with both PBIS and TIPS. 


Feedback on OLM platform 


Strengths 

Throughout our focus groups, participants provided feedback around their experience using the 
TIPS online platform (see Table 2). Feedback ranged from specific technical challenges with the 
online interface to broader ideas for how the online training could be tailored to meet specific 


Table 2: Examples of Participants Feedback related to Online Platform. 


Focus Feedback 


Strengths “It's nice that document links are ready to access and download right away.” 
“Activities were good! | liked the variation of types of responses.” 
“Lots of features that are great in online applications. It's nice that you can speed up presenter’s 
speech, jump to different points in the module and have them labeled.” 
“| found it helpful to have the pop-ups with definitions for key terminology.” 
Improvements “The navigation was fairly straightforward, but occasionally “clunky” with buttons at the bottom that 
didn’t jump out.” 
“The precision statements listed the time, but not in a recognized format.” 
“The interface for tasks was at times unclear; It would say “Great job! You did it!” after no 
meaningful response.” 
“The full screen function did not work properly. It cuts off the lower half of the video.” 
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school site and team needs. Participants shared how the online training would be useful for both 
individual new team members and an entire team learning together. Regarding technical aspects, 
participants enjoyed features that enabled them to speed up instructional videos and to review 
previous sections of the module. In addition, they felt having multiple question formats, such as a 
drop-down list or written out answers, enabled them to engage with the material in a variety of 
ways. Throughout both focus groups, participants appreciated the accessibility of material through 
both the ability to review content-specific terms through pop-up definitions as well as the inclu- 
sion of research articles, which participants could download for later reference. Both of these fea- 
tures were highlighted as being particularly useful for new team members and as helpful in 
making content accessible that generally is not available for most teachers and school personnel. 


Areas of improvement 

Some of the challenges with using the online platform that arose during our focus groups included 
specific technical errors and some difficulty around more interactive and complex responses. It 
should be mentioned that different participants had various levels of competency with technology 
which may have mediated some of their experiences. For example, some participants appreciated 
being able to speed up instructional videos, while others didn’t know how to use this function. Some 
technical errors were identified, and participants provided feedback around esthetic concerns to 
improve usability, such as having navigation buttons in a prominent color so user can more quickly 
access them. Participants reported some difficulty around providing more complex responses so that 
answers could be accurately identified by the online program. Although the platform was able to 
detect multiple components of written out responses, there were still some unclear aspects with this 
type of response that caused frustration for participants. A final area of feedback that arose from the 
focus groups was having more ways to track progress. Participants suggested enabling a feature to 
either track time spent taking the module or having a certificate of completion at the end. They 
believed this would both increase a feeling of accomplishment amongst users and be a useful tool for 
staff to report their own professional learning back to their school administrators. Participants also 
proposed the use of a chat feature to connect either staff members working on the module concur- 
rently or for staff to connect with live coaches when questions arise throughout the module. 


Discussion 


In this article, we report the process and outcomes of gathering feedback to adapt traditional in-per- 
son training to a series of OLMs differentiated for the participants to maximize utility, efficiency, and 
effectiveness in meeting the needs of key stakeholders. Focus groups were conducted with teachers, 
administrators, and specialist to gather feedback on specific content as well as how the OLM could 
be tailored to meet specific school site and team needs. The current project is an extension of existing 
research which demonstrated that in-person traditional training resulted in improved data-based deci- 
sion-making for teams. Online training has the potential to provide working professionals with flexi- 
bility to fit in with busy lives and restricted budgets (Bahr et al., 2019; Brennan et al., 2019; Dash 
et al., 2012; Davis, 2009; Glover et al., 2018; Stanford-Bowers, 2008) we are engaged in the process of 
shifting the effective in-person training to an online platform. Creating OLMs and other professional 
development materials should be an iterative process of cycles, revisions, and progressive refinements 
(Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003; Collins, Joseph, & Bielaczyc, 2004; Glover et al., 
2018; Seago, Jacobs, & Driscoll, 2010). In this study, we followed this iterative process of development 
of the TIPS OLM by engaging with educator stakeholders. 

The results from the focus group sessions indicate that members of Tier 2 behavior support teams 
found the online content and method acceptable, perceived it as having a high degree of utility, and 
were able to identify meaningful areas for improvement. Comments from both groups were generally 
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positive and users expressed excitement regarding the potential of online training and its manifest- 
ation as presented (see Tables 1 and 2). Further, users from the first focus group provided feedback 
regarding specific items or technical issues which our team was able to incorporate prior to the 
second focus group. Feedback related to these addressed issues was positive during the second focus 
group, and sometimes identified as a strength of the training. Users identified an essential tension 
between the convenience and power of electronically delivered instruction in terms of its potential 
for individualization and flexibility, and the benefits of face-to-face instruction with rich interper- 
sonal interactions which are integral to well-run traditional professional development activities. 

Users also indicated that they believed the strengths of electronically mediated training were 
greater than the potential drawbacks of losing some of the interaction they found useful in face- 
to-face settings. Specific supportive feedback included requests for a chat feature and expressions 
of enthusiasm related to the existing interactive practice activities. There was wide variability in 
the technical proficiency of users engaged in the focus groups for the present study, and this 
appears to have been related to the scope and content of feedback received regarding the tech- 
nical aspects of the online platform. All participants were able to work through the training seg- 
ments with a reasonable level of independence and all expressed excitement for the availability of 
online TIPS training for their school teams. 

Our work was well connected to effective professional development practices (cf. Darling- 
Hammond et al., 2017a, 2017b, 2017c). We focused on a singular content-based practice (i-e., 
problem solving) and used engaging materials to provide contextualized learning with coaching, 
progress monitoring, and feedback. Findings from focus group participants supported the initial 
development of our OLM and provided useful directions for finalizing, expanding, and imple- 
menting the work. 


Limitations 


The focus groups were conducted with school personnel from one school district in the Pacific 
Northwest with limited racial and socioeconomic diversity using a small sample recruited through 
purposive sampling. Studies conducted with key stakeholders across multiple school districts with 
a more representative sample will provide additional information about professional development 
needs across different groups within schools (e.g., first year teachers, support staff) as well as the 
best modalities for delivering such trainings. This was not an outcome study; in fact, we sought 
to gain insight and guidance from educators on the development of an online learning platform 
that would make online professional development experiences more beneficial, interactive, and 
salient. As such, the feedback provided was not collected to document summative performances, 
but for the formative purpose of improving training opportunities and tools for key stakeholders 
involved in educating and supporting students. 


Implications 


It is necessary for researchers to share iterative development procedures so that we can continue 
to increase the rigor of the development process as it pertains to online learning for teachers. A 
strength of our study is that the TIPS face-to-face professional development has been shown to be 
effective in previous empirical studies (Horner et al., 2018; Newton et al., 2012). This connection 
between previously conducted experimental studies and qualitative development projects could be 
used as a model to develop and disseminate high quality online professional development opportu- 
nities. This study demonstrated that teachers want their online learning to be differentiated based 
on role (ie., special education teacher, principal, general education teacher) so that the time they 
spend learning is meaningful and as efficient as possible. Recent research has demonstrated that 
targeted and online learning opportunities have the potential to make evidence-based practices 
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accessible to a wider range of teachers who have increasingly demanding personal and professional 
schedules (Brennan et al., 2019; Glover et al., 2018; Simonsen et al., 2019). 

Future research should explore and empirically evaluate selective critical features of profes- 
sional development to identify the most effective ways of translating training materials into prac- 
tices used within the classroom. Additionally, evaluations documenting school-based changes and 
outcomes resulting from this type of professional development are needed and clearly warranted. 


Conclusion 


Educational research, and more importantly, educational intervention and tool development benefits 
from the input and guidance of the individuals charged with using the tools in their schools. As 
such, collaboration between intervention developers and school personnel is necessary for developing 
comprehensive supports that fit the school context, are feasible to use, and render the outcomes that 
are needed in schools. Although studies such as the one presented do not provide empirical evidence 
for key features of effective professional development, they are a necessary step in the development 
of supports that will ideally be used by the very stakeholders that helped develop them. 
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Appendix A. TIPS EdTech Focus Group Guiding Questions 
Before the Module 


We are developing online learning modules (OLMs) for TIPS training. Before looking at the first module, we want 
to know... 


1. What are some things you like and/or dislike about online learning? (you may find that some of the things 
you dislike are in the module we show so it will be good to reflect on that) 
2. Give information about what they will be seeing: content, prerequisite 


After the Module 


We need your help with the form (how does it look), function (how does it work), structure (how well was content 
presented), and other areas of the OLM ... 
Form (How did it look?) 


1. What did you like about the OLM? 
a. Organization 


b. Flow 
c. Navigation 
d. Other 


2. What was confusing and/or needs improvement in the OLM? 
Function (How did it work?) 
What parts of the OLM (e.g., activities, feedback) should be increased or decreased? 
4. What was confusing and/or needs improvement in the OLM? 
Structure (How well was content presented?) 
What content (e.g., focus areas, examples, practice items, assessments) needs more or less coverage? 
6. What content (e.g., focus areas, examples, practice items, assessments) was confusing and/or needs improve- 
ment in the OLM? 
Conclusion (What else do we need to know?) 
7. How did going through the OLM compare to your previous training experience with TIPS? 
a. Was the organization and flow similar to live professional development? 
b. Was the content presentation more or less engaging than live professional development? 
c. Were the activities more or less engaging than live professional development? 
8. What else should we know moving forward? 
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